Building “Stuff”?

Charles F. Vardeman II

Center for Research Computing, University of Notre Dame

2023-09-05

Building Stuff?

Building Agents based on Large Language Models!

Building “Stuff”…

Building “Stuff”…

You are here ⬇️

What Kind of LLM Agents are we trying to build?

  • Conversational Agents
  • vs. Cognitive Autonomous Agents
  • vs. Agents tuned for a Data Processing Task

We will focus on Conversational Agents…

The Best Advice we can Give

Caveat: You are at the Edge of Research and Practice!

Instruction Prompting

(GPT-3) Instruct-GPT Reinforcement Learning from Human Feedback

Facilitates Conversational Agents to “Converse” in a Set Style!

(GPT-3) Large Language Models are Zero Shot Reasoners (Chain-of-Thought Reasoning)

Tools for “Prompt Engineering”

Tools for “Prompt Engineering”

LLMs as Reasoners using Prompts!

Prompt Enginnering Guide

We want Large Language Models to be Factual!

  • “Fine Tuning”
  • Retrieval Augmented Generation (RAG)

We want Large Language Models to be Factual!

  • Fine-Tuning: augment the behavior of the model
  • Retrieval: introduce new knowledge to the model
  • Retreval Aware Training (RAT) Fine-tune the model to use or ignore retrieved content

LlamaIndex to Build Hybrid KGs

Gorilla: Retrieval Aware Training for APIs

Gorilla: Retrieval Aware Training for APIs

“Big Models” vs “Small Models”

  • Models as a service (Bedrock, OpenAI API, Anthropic Claude)
    • Generally more difficult to “Fine-Tune” (GPT-3.5 turbo)1
    • Models are generally more capable (Factuality, Instructions, Reasoning)]
    • “Coin-operated” pay per/token
  • “Open License” 7B-70B Parameter Models
    • Mostly based on Meta AI LLama or LLama 2 models
    • Require more effort to work consistently
    • Models can run on reduced hardware requirements
    • Can be fine-tuned for task specific workflows

Small Models with custom grammar (llama.cpp)

JSON-Grammar

root   ::= object
value  ::= object | array | string | number | ("true" | "false" | "null") ws

object ::=
  "{" ws (
            string ":" ws value
    ("," ws string ":" ws value)*
  )? "}" ws

array  ::=
  "[" ws (
            value
    ("," ws value)*
  )? "]" ws

string ::=
  "\"" (
    [^"\\] |
    "\\" (["\\/bfnrt] | "u" [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F] [0-9a-fA-F]) # escapes
  )* "\"" ws

number ::= ("-"? ([0-9] | [1-9] [0-9]*)) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? ws

# Optional space: by convention, applied in this grammar after literal chars when allowed
ws ::= ([ \t\n] ws)?

“The state of GPT” Recommendations

Open Source Community

Today is the First Step on your Journey to Building LLM Based Applications!…